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PREFACE 


This paver presents the author's attempt to develop a short 
range forecast verification for Navy usee It is intended to be a 
Simplo practical system vet based on sound moteorological and sta~ 
vastical principles. 

Undertaken as the thesis requirement for tho degree of Master 
of Science in Aerology, this paper was prepared at the United States 
Haval Postgraduate School, Monterey, California, during the academic 
year 1949-1950. 

The author is deeply indebted to Professor We De Duthie, ror the 
original suggestion of the subject and for his valuable assistance 
during the devolopment. He also wishes to acknowledge the assistance 
rendered by Associate Professor Ae Boyd Newborn in the preparation of 


the verification tolerance scales. 
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Ie INTRODUCTION 


The subject of forecast verification continues to be as contro- 
vorsial now as it was when the pioncering work in the field was done in 
the last century. The variety of opinion and counter opinion still exists, 
as there are no authoritative criteria for settling an argument of such a 
subjective nature. However, differont systems will be required to fit the 
diverse uses of forecasts» 

In this paper the requirements of a system which meets the particular 
needs of the United States Naval Aerological Service are set upe Tne various 
verification schemes proposed and used in the past are examined. The simple 
percentaze of correct forecasts is rejected for several reasonse The need 
for some logical basis of comparison is thon establisheds 

Three classes of these bases for comparison are examined. They are 
climatology, pure chance and allied computations, such as skill score, and 
persistence. Persistence is established as the most logical practical basis 
for comparison for the Navy purposes in verification as set forth heree In 
fact, the first two schemes of comparison, climatology and chance, are 
positively rejected as unsuited for either theoretical or practical reasons 
or both. Climatolofical evidence is presented in support of the above 
contention.» 

Then with the basis for the verification system decided upon as a 
comparison with persistence, the details are developed in accordance with 


the laws of probadility and statistics. 
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Before an objective verification can be attempted the torms in 
which the forecast is to be made must be rigidly defined. The exact 
form of the 36<hour forecast is thereforo specified. Some features of 
the present Navy forecast form were retained but many innovations are 
made which should incroase the preciseness and completeness with which 
the forecast is stated by the forecastcre 

The development of the tolerance tables is then presented. The 
desroe of correctness of the forecast is oxamined with each meteorologi-~ 
cal element considered separately. Differences with the present llavy 
system are evident here, Precipitation, cloud cover, and visibility 
are each considered separatelys 

The verification score is determined for each of eight meteorological 
elements forecast, vize precipitation, average cloud cover, lowest ceiling, 
lowest visibility, surface wind direction, average hourly wind velocity, 
maximum single gust, and maximum or minimim temperature as appropriatee 
The score depends on two things, amount of change since the previous day 
and the degree of correctness of the forecast. The sradations of the 
tables in degree of correctness are matched with the limits of observerable 
or predictable accuracye The logical basis for using the amount of change 
from the weather of the previous day, ieCe, persistance, as a criterion of 
difficulty of a forecast, is established. The numerical score obtained by 
adding the scores for each forecast element, read in the appropriate toler= 
ance table, has no percent significancee By taking a score of zero for a 
correctly forecast persistant oceurrence, the superiority or deficiency of 


the forecast compared with persistance is automatically establishede 


(2) 
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The verification system is then tested on a series of forecasts 
for typical Navy stations, illustrating tho similar total scores ob- 
tained from the verification of forecasts ranging widely in difficultye 
Finally, the proposed forecast form and verification systom are 
compared with the system now in use by the Naval Aerological Services 
The preciseness of the terms in which the forecast must be stated in= 
creases its value to the consumer and the time required for verification 


compares favorably with the present systeme 


(3) 
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Il. REQUIREMENTS OF A VERIFICATION SYSTEM FOR NAVY USE 


In order to devise a forecast verification system for Navy use, it 
is first desirable to enumerate the uses of verification of forecastse 
From these uses, the particular qualitios most desirable can be derivede 

The principal uses of verification should be in studies by which 
forecasts could be improved. A study of errors may indicate consistent 
trends toward too radical or too conservative forecasts of particular 
weather elements. 

Another widespread use of verification is in determining the maximun 
period for which forecasts are of value by establishing a significant mine 
imum score. Also forecasters are frequently ranked in ability according 
to their accumlated forecasting averaroe 

These uses in themselves may not appear to justify the time and 
enorgy required in verification, It is felt that accurate verification 
records will provide incentive for individual forecasters to increase 
their skill while the absence of verification will have a contrary effect. 

These rather general uses of verification require several restrictive 
specifications when considered from a Navy viewpoint. Any verification 
system will inevitably be used to rank forecasters at one station and 
possibly among several different stations. The system must be based upon 
and developed by sound statistical principles, It is desirable also that 
the score obtained by verifying the forecast reflect only the skill of the 


forecaster. This reauires a system that will give comparable scores at 


(4) 








one station for forecasts made in a variety of synoptic situations of 
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III. TYPOS OF VERIFICATION SYSTDIMWS AITD THEIR FISTORICAL BACKGROUND 


Completeness and continuity in the historical development of veri- 
fication systems and their classification is largely made possible by the 
excellent survey of the literature by Muller [ 5 ]- This is especially 
true of the discussion of the contributions of European meteorologists 
prior to 1920. 

The earliest verification systems were based on a simple computation 
of percentage of correct forecasts. This system is referred to as the 
percentage system, <A forecast was judged to be a completo success (hit) 
or a complete failure (miss) according to fixed tolerances. For example, 
with a tolerance of 4 degrees, a forecast of any temperature 41 to 49 
derrees inclusive would be judged a complete and equal success if the 
observed temperature were 45 decrees. Any other forecast temperature 
would be scored a failure This is the percentace system with fixed 
tolerances. This rather naive method is still in use by some weather 
services although the meaninglessness of percentarce of hits as a test 
of skill in forecasting was pointed out by Koppen [4] as early as 1906. 

Modern users of this method goon ceites Boies scales, assigning 
points according to the degree of success of the forecaste. These ratings 
are then converted to percentages. The Naval Aerological Service is one 
of the very few still retaining simple percentare verification with fixed 
tolerances. 

The second group of verification systems includes those in thich the 


score is determined by the degree which the success of synoptic forecasts 


(6) 


differs from the success of some type of comparison forecast, This 
eroun is divided into types accordins to the kind of comparison forecast 
usede These types involve. (a) elimination of pure chance; (b) comparison 
with somo kind of persistence forecast; and (c) comparison with some kind 
of climatolocical forecast. 

K6ppen's original suggestion was a comparison with random forecasts. 
This would eliminate the portion of the forecast's success due to chancee 
The current use of skill score is another example of the elimination of 
chance successes» However, the use of skill score is limited to variates 
capable of representation in a tetra-chloric distribution, such as those 
which can be analyzed on a simple occurrence or non=occurrence basiSe 
This type of analysis applied to thunderstorms or precipitation or other 
discrete weatner phenomena is very effectivee 

A second type of comparison forecast was developed by Yeidke [ 3]. 
He accepted Kdéppen's view that percentage of hits is an inadequate measure 
of forecasting skill. He proposed a system based on using the previous 
day's weather; i.6e, persistence, as the comparison forecast. The actual 
verification score was obtained from complicated formulae. An interesting 
variation of this method was proposed by Dinies } 2] for use in the German 
Weather Service. The observed weather of the particular day in the previous 
year was used for comparison. 

More recently a tnird type of comparison forecasts has come into linited 


usé@e These systems use climatological records as a point of references 


Clayton's [ | extonsive work and the method proposed by the Verification 
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one author included has published articles expressinz the belief that 
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forecasts should not be verified, Schmauss [6 _ Although there is 










some modern support for this theory, certainly the great majority of 


meteorologists recornize the need for objective forecast verification, 
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IV. SELECTION OF TYPE OF VERIFICATION SYSTEM BEST FITTED TO NAVY USE 


The first and simplest type of verification system is based on a 
calculation of the percentase of correct forecasts, with each forecast 
of an elemont being considered as a hit or a miss according to a fixed 
tolerance table. This system is presently used by the Naval Aerolorical 
Service but it has soveral serious faults when avplied to one station and 
even more when applied to a variety of stations, 

Forecasts vary in difficulty from day to day and from season to season 
at the same station, Thus forecast scores computed on a basis of percentase 
of hits will show wide variation from day to day, with seasonal trends probe 
able, all independent of the skill of the forecaster. 

The complexity and rapidity of weather chanses which,in reneral in~ 
crease with increasing latitude and vary with georraphical location,will 
make impossible any comparison of forecasting scores from different 
locations such as the widespread Navy stations. 

The tolerances used to determine whether a forecast is a success or 
failure in the percentare systom are not suitable for stations with wide 
meorraphical differences. For example, with a tolerance of 4 degrees 
Fahrenhoit in daily mininum temporature at a maritime low-latitude station, 
100 percont hits might be possible on a repeated forecast of one particular 
vemperature, while at an inland hign-latitude station, rreat forecasting 
Skill would be required to obtain a score of 75 percent hits on mininum 
temperature for the same month. 

Thus a sysvem employing percentage of correct hits seems extremely 
unlikely to fulfill the Navy reauirements as outlined in Chapter II, even 


with considerable modification. 


(9) 
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A systen emplozyinr some type of comparison as a referonce point 
and making some allowance for forecast difficulty seems necessarye 
The elimination of tho successes achieved by nure chance is not adequate, 
since it does not change with changins forecast difficulty. Uxnectancies 
calculated on the basis of climatolov;y seem most logical, Then the correct 
forecast of a rare event could be given hicher weisht than the correct fore- 
cast of a common evente 

An attemnt was made to set up such a system. Records were obtained 
from several typical avy stations by a the various weacher elements 
as they appeared on syvmontic charts for several months of all seasons of the 
year, The variations in averase values and distribution of the vars.ous 
weather occurences were so sreat that tolerance scales for forecast score 
computed on the basis of long-term averarves would have little meaning when 
applied to an individual soason or monthe For example, advection fog, at 
Pensacola had a long-term average occurrence of oight 12-hour periodse 
The best estimate of the expectancy of fog at Pensacola for any 12-hour 
period would be 8/566 However, if this frequency were used in scoring 
verification of fos forecasts in February 1947 and February 1948 it would 
have little meaning, for in February 194] fog occurred not at all and in 
February 1948 it occurred during twenty-three 12-hour periodse 

Temperature records are cited in Table le These data show the 
variation of monthly mean temperatures about the lonr-term mean. This 
tabulation also shows that the variation of monthly means is so sreat that 
tolerence intervals based on long-term means will be too larse to be 


practicale 
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INTIR DIURNAL TEMPERATURE VARLATIONVS IN DEGRESS :AMRERRSIT 


~B8 =f -6 =4 92 O 2 4 6 § 7 & 
1930 2 i 2 & 2 7 i 1 2 1 
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L9e] l L L a 6 y ? > ab 2 1 
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VARIATION OF MONTHLY MEAN TEMPDRATURES ABOUT LONG TERM MEAN 
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I'requency and probability of occurrence of the various woather 
elemonts computed after the forecast interval (month) is over would 
provide a lorical basis for assessing difficulty. However, this is 
considered impractical, The forecaster should have prior Imowled<:o 
of the verification tolorancese Aerological office routine makes a 
daily verification desirablee 

The remaining basis for comparison of forecasts is persistencés 
This is less desirable theoretically than probabilities computed post 
facto, but is much more practical in applicatione Day-toeday persis~ 
tence was chosen as being the most probable as well as most convenient 
standard. Persistence has the freat advantare of universal application 
to any climate. The fact that the most probable weather for tomorrow 


is that which occurs today is supported by the data of Table 2, 
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Ve DEVELOPMENT Cl TOLERANCE TARLOS AD SCOhI8G SYSTEM 


Having selected persistonce as the basis for comparison, the details 
of the vorification system remain to be decided upon. Several principles 
have been solected as euides in the development of the verification schemes 
for the various weather elements of the complete forecast. These aro bricf- 
ly discussed here along with the specific purposes they serves 

In the forecast and verification system presently used by the Naval 
Aerological Service, the method of presentation of state of the weather 


is considered particularly weak. A prescribed set of terms aro available 


ct 


in this system and one of these must be chosen by the forecaster and 
entered in the apprepriate space in tho forecast forme These torms range 
from definite events, such as thundershower and showers to very indefinite 
terms, such as "mostly fair" and "threatening". These last two terms are 
defined as a certain range of cloud cover with a small amount of precipi- 
tation permitted. Fog (defined as visibility below a certain mininum for 
a certain duration) is another permissable terme Thus in this system, 
precipitation, cloudiness, and visibility are all considered in one fore~ 
cast element, the state of tne weather. The result can only be confusion 
and uncertainty in the minds of those who must use the forecaste 

This difficulty will always be present when a word or short phrase is 
used to summarize the weather conditions of a period of time. A maximm of 
clarity can be obtained by stating the forecast for each weather element 
(precipitation, cloud cover, visibility, etc.) separately. The aaa sum=~ 


mary spaces of the forecast form must be filled with specified terms whose 


meanings is clearly defined in the minds of the user as well as the forecaster. 


(13) 
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In the proparation of the tolerance allowances of tno proposed 
svstoen it is assumed that the most probable occurrence for any forocast 
weather element for the following day will be tho observed weather of tho 
present day. It is further assumed that the distribution of tomorrow's 
weather values about today's is normal. This is supported by the data of 
Table 2. Although only temperature data are oxhibited, shorter tests were 
run with other meteorolorical elements and the results were cormarablee 

The forecast elements are senarated into three classes according to 
their type. Frecipitation is analyzed on an occurrence or noneoccurrence 
basis. Wind direction, cloud cover, and temperature are continuous vare 
lates with no variation in meteorological importance. That is, a North 
west wind direction is not considered any more or less important than any 
other wind directione The third class of variates includes ceiling, visi- 
bility, end wind velocity which, althourh continuous, have verying impor- 
tance. For example, with low visibilities an error of one nile is more 
important than the same error with 15 mile visibility. The change in 
importance was considered from a meteorological viewpoint. lo attempt 
was made to assess the operational importance of various forecasts, This 
would vary widely according to the type of military operation for which the 
forecast is intended. For example, for hich level pnhotorraphy an overcast 
at 5000 feet might be considered bad weather, while for surface ship 
operations it misht be quite wmimportante 

In all tolerance tables an attempt was made to verify to the same 
accuracy as the observation is made. ‘when the observed maximum temperature 
is 659, a forecast of 65° should set more credit than any other forecast, 


with the score decreasing rapidly with increasing errore 


(14) 
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A fundamental question remaining is the scorins syston. ‘hat rasio 
should be chosen between the credit allowed for a correct persistence 
forecast and the credit allowed for a correct change forecast? 

The persistent weather of low latitude stations is reflected in the 
high forecasting scores obtained there with fixed-tolerance percontago 
verification, The rapidly chanrins weather of higher latitudes causes 
lower scores when forecasts are rated on the same system. This inequality 
is independent of forecast skill and should be removod, 

After a study of percentace verification records from high and low 
latitude stations a ratio of six to ten was chosen. This value was comnuted 
from the averare percensace scores at stations in both tynes of seosraphical 
location. 

The tolerance tables employ these ratios. The decrease in credit with 
increasins error is computed from ordinates of the normal curve. Using a 


hypothetical forecast element as an example, the scoring for errors fron 


O-4 units from the normal ordinates would be: 


Error in units Score 
0 LO 
1 946 
2 8 
a 5 04 
4 > 


In order to avoid fractions the scores are doubled and rounded off to 
the nearest whole number according to standard rounding procedure. The 


above example then becomes: 


Error in units Score 
O 20 
iL L9 
2 16 
3 Li 
4 6 


(15) 
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he abscissae in the tolerance Tattle are iittel b¥ the inverge 


normal ordinate according to tie aniount o. chause from nrevlous Gave 


In the hyrothetical examle, the zero error row micht rai acrgss: 


Difference from yesterday in units 
error 0 at z > 4 


0 12 13 14 17 20 


Then the commlete table for this simplified illustration would be: 


Difference from yesterday in units 


Beror 
in units O ve (2 > 4 
0 le 155] 620 
1 6 7 10 16 19 
2 0 Sugit 16 
3 Oe) 0) OF 7s 11 
4 0 0 0 4 6 


The voint at which tne forecast has zero value is arbitrarily 
selected for each table. Twoeway tables such as the ore above are used 
to verify wind direction, cloud cover, and temperatures The score is 
determined by tno error in the forecast and the persistence of the element. 
In the cases of those elements with varyinc natural importance a 
three-dimensional table is used. This requires two plare tables which de- 
termine forecast score from three variables: the meteorological importance 
of the forecast, the amount of persistence, and the error in the forecast. 
The score for a correct persistence forecast is now translated from a 
value of twelve to zseroe This serves two purposese The correct nersistence 


forecast automatically scores zero, making comparison possible without 


(16) 
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computation. Also it removes any possibility of tho final score boing 
confused with a percentare score on tho basis of 100 percent for a 
correct forecast, Several objective verification systems in the past, 
notably those of Heidke and Claycon, have boen considerably criticized 
by othor forecasters solely because the numerical score computed by their 
system was small compared to 100 percent. 

After this translation of reference point, the exemplo above would 


appear as follows: 


Difference from yesterday 


Error 0 1 2 4 4 
0 om 25 8 
1 6 & = 4 7 
2 mo ae OS] ek C4 
3 ie 2 clea ah axl 
4 a2 «el? «nl? ae “Was 


Special details of the scoring system as applied to the individual 
forecast olements are discussed herée 
le Precipitation. The type or intensity of precipitation is of little 
importanco for Naval uses when tne effect on ceilings and visibility is ade- 
quately forecast. Quantitative forecasting of precinitation is beyond the 
present precision of the science, For those reasons precinitation forecasts 


are analyzed on a simple occurrence or non=occurrence basise 
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The only modification introluced is to cover cases in which only 
very small amounts of precinitation occurrede In such cases a forecast 


ae 


of no vrecinitation would be given half credite Amounts up to and in- 


cluding .02 inches are defined as slisht procipitation for this special 
modificatione 

2-0 Cloud Covere lo special details, 
4s  Ceilinge The meteorological importance is assicned roushly in ace 
cordance with the Civil Aeronautics authority requirements for instrument 
and contact flights. The error is assessed in 500 ft. wnitse 

4. Wind Direction. An cisht-point compass was considered adequate for 
seneral meteorological uses. <Any forecast with an error of more than two 
points was considered worthless. 

5e @«=©©riiind Velocity. Although average velocity is computed to the nearest 
kmot,. the use of a twoelmot interval in computine error was necessary for 
brevity. The increased accuracy attained oy the larger table necessary to 
include one-kmot intervals is not justified. The change of sccres would be 
Siznificant in only one or two places in the entire table. 

6, Maximum Gust, The present verification system verifies maximum hourly 
velocitye This value has little meaning, so maximum single cust was sub- 
stituted. It is forecast and verified to five-’mot intervals. 

Je Visibilitye The meteorological importance was acain decided in ace 
cordance with requirements Tor CAA Closed, Instrument, and Contact flisht 
rosulations e 

8. Maximum and Minimum Temperatures, Verification tables have tywo-derroe 
intervals for brevity although temperature is observed to the nearest whole 
degree. The small errors occurring where tho slope of the normal curve is 


great and the difference in scores between spaces in the table relatively 


large are discussed under wind velocity abovee 
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VIs PROPGSED FORECAST [ORM 


Before any objective verification can be attempted, the forecast 
must bo stated in precise meanineful terms, The specifications for each 
forecast element require a complete concise statement of the expectod 
weather with no chance for vague terms or "hedges", The practice of 
hedsine in forecasting is old and widespread, It is larcely responsible 
for the popular misunderstandinzs of the possible achiovements and also 
the limitations of the sciences In this proposed verification systom the 
maximm score is attained only by a very accurate forecasts 

With forecast systems that permit indefinite complicated terms such 
as "mostly fair", there is a strong tendency for forecasters to attempt to 
include in thoir forecast all possible synoptic develonmonts. ‘when fixed 
tolerance systems are not clearly thourpht out there may exist favored 
mumerical forecasts. For example, in the system in current Naval use the 
tolerances change in too large intervals. Thus with an observed averare 
wind velocity of ten kmots the allowance for a success is plus or minus 
two lmots while the allowance for a hit with an observed average velocity 
of eleven knots jumps to four imots. This favors forecasts of nine or 
eleven kmots and makes the use of ten !mots penalize the forecastere 
Numerous other cases of this inconsistency exist in the current systeme 

In this proposed forecast and verification system, the forecast is 
required to be stated in simmle precise fashion and the verification 


scoring does not encourafe any attempt to hedge. 
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The present box system of NavAer 447(a), containing a brief statemont 
of the values to be verified is considered desirable. ‘these boxes contain 
the forecast in explicit unambizuous terms, readily available for verifie- 
catione These boxes are arranced in a vertical column with the detailed 
forecast opnosite theme 

The forecast form surgested here is for a l2-hour forecast interval. 
Three of these forms would ordinarily comprise a normal forenoon forecasts 
However, two forms for day and following nisht could be used for a prelini- 
nary (early morning) forecast and the lenrth of the forecast period could be 
extended by addinse more of the basic l2-hour forms, 

Figure 1 is an example of the l2-hour forecast form. [Tisure 2 is the 
same form containins a sample forecast, The specifications for the forecast 
are set forth belowe 

Precipitation. Box: yes or no as appropriate, Detail: if box contains 
yes specify type or types of vrecipitation, time of besinning and ending if 
within forecast period, intensity and changes in intensity, and amount 
of precipitation expected in periode The special classification, light 
precipitation, is intended for use only in forecast verificatione 

Sky conditione Boxes: state averase cloud cover in tenths for neriod 
in upper box. In lower box specify lowest ceiling expected to occur for two 
successive observations. Detail: specify chronological variation of cloud 
cover. State maximm and minimum number of tenths expected and sive times 
of occurrence. Specify chronolorical variation of ceilings, including high- 
est and lowest ceilings expected and times of occurrence. Include statement 


of turbulence and icing when anplicable. 
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Visibility. Box: lowest visibility in miles expected te occur for 
two successive observations. Detail: specify chronolorical variation of 
visibilitye 

Surface winde Boxes: top box for prevailins wind direction during 
period. Middle box for averase hourly velocity in knots, Lower box for 
maximum sinzle cust in inotse Detail: specify all significant changes in 
direction, velocity or gustiness. Include hirhest and lowest hourly aver- 
ares expected with times of occurrences 

Temperature, Box: maximum or minimum temperature in degrees Fahrenheit 
as applicable. Dctail: specify variation of temperature by siving forecast 
temperature for 4-hour intervals during period. Include time of maximum or 
minimum, and where applicable, the time at which the temperature is expected 
to reach freezing point (32°F) whether rising or fallinge 

Winds aloft. Wot verified. Forecast for two times to coincide with 
local pilot balloon observations during forecast periods Are to be forecast 
for 6 levels te be chosen by the forecaster according to his operational 


commitments e 
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FPige 1 PROPOSED FORECAST FORM 


Local Time Forecast Hffective 
Precin Precipitation 
Yes No 





Ave eCloufi Cover Sky Condition 
Tenths 


Lowest Ceiling 
Hundreds| of feet 


Lowest Visibility Visibility 
tiles 





Surface Wind 
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WAX e Min 


oF | 


emperature Temperature 


Winds Aloft 
Time 


Level 1 |Dir=VYVel ae 4 {DireVel ‘Level 1 |Dir-Yel ‘Level 4 Dir-Vel 
nash — 5 |Dire-Vel tewel 2 pir=vet Love 5 |Dir-Vel 


— a 


| 
ee eet 
Level 3 |Dir-Vel |Level 6 |Dir-Vel |Level 3 |Dir-Yel Level 6 nir<Vel sail 
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Fise 2 EXAMPLE OF TWELVE HOUR FORCCAST ON PROPOSED FORM 


Forecast Effective 1800-0600 Local 


Precip aa Precipitation ~ 
ae Lisht showers accompanying weak cold frontal passare at | 


2200 locale Showers occurring 2000 local to 2400 locale | 
Precipitation 10 inch expected. | 
| 











Sloy Condition 

Five tenths small cumulus clouds based at 5000 fte at 
1800 local increasing to overcast with showers. Ceil- 
ings lowering to 2000 ft. in showers. Tops 4500 ft. in 
creasing to 7000 ft. in showers.Clouds flattening and 
decreasing to five tenths strato-cumulus base 2500 ft. 
tops 3500 ft~. at 0600. Light to moderate icing in 
clouds above 4000 ft. after 2400. Lisht to moderate 
turbulence during frontal passaree 


a ee EE Oa 


—_—. = 








Vasibility 
% Miles) Visibility 7 miles hazy at 1800, decreasing to 
| 4 miles in showers 2000-2400 and increasing to 


15 miles at 0600. 









Surface Wind 
Surface winds SW10 - 15 Paaets with gust to 20 knots 


Ave GiindlVel veering to INl with crusts to 30 knots at 2200 local and 
|S" )=—s agereasine slowly to NT 8-12 mots at 0600. Highest 


» 
on —é ee ene ™ 


| We | hour 15 Ikmots 2100=2200. Lowest hour 8 kmots 0500-06004 
ax eGust Maximum gust expected about 2200 locale 
|__30 | 





rs i Temperature 


i 


1800 local 49 degrees 2200 local 46 desrees 
0200 local 43 derrees 0600 local 37 degrees minimum 


i 

; 

: 

2100 local winds taloft 0300 local re : 


/ 2000! | 210-30 .15000' 240-45 | 2000! [310-22 | 20000 | 245-40 
| —. 7 


| 5000! 220-55 120000! peep | 5000! 250—25 P ae 25 5=50 
- 235 =—355 anny 260265 E 1LOOOO!? |240=40 | 40000! | 270-85 
ee : L 7 


Ons em oe. oe See — ee ee ee ee Oe 
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1, 


De 


4 


De 


Precipitatione 


Cloud Covere 


Wind Directions 


Maximum or Mine 


inter Table 43 if forecast is corroct and record scorée 
in avorare cloud cover in tentns is recorded for each 
houre A maximum of ten tenths is permitted. The averag 
of these values is used for verification. cnter Table 4 
with the crror of the forecast in tenths and the chanre 
from yesterday in tenths. Record this score. 

Record a direction to tho nearest of eifht points fron 
tne record of the selsyn recorder. Use direction which 
prevails for greatest number of hours. Enter Table 5 
with error of forecast in points and the difference in 


direction from yesterday in points. 


imum Temperature. Obtain maximum or minimum temperature to nearest whole 


Averace Hourly 
Wind Velocity. 


degree from hourly and check observations and/or maximum 
or mininum thermometer or thermograph if accurate. Inter 
Table 6 with error in derrees and difference from yester- 


day in desrees.e Record score. 


From single or multiple resister record total number of 
knots passing anemometer during forecast period and obtain 
averasce hourly velocity to nearest whole number. Enter 
Table 7 with this velocity and yesterday's average velocity 
and obtain indicator letter. imter Table 8 with indicator 


letter and error in mots and record score, 
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6-¢ Maximum Guste From Selsyn Recorder record of bridled anomemeter obtain 
maximum single cust during period. Unter Table 7 with 
this velocity and yesterday's averaso velocity and obtain 
indicator letter. Enter Table 8 with indicator letter 
end error im ‘mots and record scores 

T+ Ceilinge From aimmys record, special, and check observations 
obtain lowest ceiling occurring for two consecutive 
observations during the forecast period. Enter Table 7 
with this ceiling and yesterday's lowest ceiling and obe 
tain indicator lettor. Enter Table 9 with indicator 
letter and error in nearest 500-—foot units. Record scores 

Ge Visibility. From record, special, and check observations obtain lowest 
visibility occurring for two consecutive observations dure 
ine the forecast period. Enter Table 7 with this visibi- 
lity and yesterda:tts minimum visibility and obtain indi-~ 
cator letter and error in miles and record score. 


Add these eight element scores to obtain final score. 


(25) 





As an example the sample forecast exhibited in Figure 2 is verified 


belovwe 


Clement 


precipitation 


average cloud 
cover 


lowest ceilings 
wind direction 
aves velocity 


MAX. gust 


mine temperature 


Previous lirsht 


none 


5 tenths 
above 10,000 ft. 
Sif 
7 


12 


43 


Forecast 


yes 


6 tenths 
2000 ft, 
I! 

12 
30 
31 


Observed 


yes 


LO tenths 
1200 fte 
SW 
14 
28 


42 


score 


16 


<12 


A NT 


- 4 


It is intended that the scores for each l2=hour period be kent 


sevarate rather than be added together or averased in any Waye The 


increasing difficulty of periods farther away from the forecast time 


makes averacing of two or more of these neriods detract from the mean= 


ime of the scores attained. 
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PRECIPITATION 


score 
Correct Change Forecast 16 
Correct no Chanre Forecast 0 
Incorrect Change Forecast 
Precipitation less than .02 inch’ ~4 
Incorrect no Change “orecast 
‘Precipitation less than .02 inch) =ii2 
Other Incorrect Forecasts n24 

TABLE 3-6 
CLOUD COVER 


Change from Previous Day in Tenths 


1 =e © 0 2 2 4.6 & 12 

Pome im «262 -12 -10.-10 @ 8 ~—6 =4 OF 4 8 10 
Tenths 

3-18 m1E =-16 =16 -16 =14 -12 -8 =4 0 2 

4 =24 -24 -24 -24 -24 ~18 -16 -14 -10 =-5 —4 

5 =24 —24 -24 =24 =24 ~24 =24 -18 ~16 ~12 ~10 


6 =24 -24 -24 -24 =24 =24 =-24 -24 ~24 -18 -=16 


TABLE 4. 
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TEMPLRAT URE 


Chanre from Previous Day in Degrees Te 








DVDICATOR LETTERS 
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Maximum Gust 


oO 


ts 


J 


L 


N 


“rror in Knots 


O 


8 


WI ONAN 


UW 


7 


? 


=2 
2 


a4 


LO 


0 


-l 


-1 


2 


-l2 


-1l2 


-12 


LP 


~al2 
=12 
-12 
-12 
-12 


12 


VERIFICATION SCORING FOR WM VELCCITY 
veraho Vol 
Srror in Enots 
O=l 2—3 485 6-7 849 
» fo 7.6 4 0 
a 
eS &@ 2 0 2 
3 2 1 era 2 
5 4 2 #1 -4 x6 
z 2 L 92 @5 | 
2 OO -3 =7 =9 12 
4 3  -2 6 =9 12 
L O =4 =7  =9 =12 
1 QO 44 #] <-9 =12 
3 Ll 3 8 ~12 12 
1 O 4 8 -12 «12 
O -1 -5 99 -12 -12 
O ~~ 6 <9 #12 ~12 


-12 


20 
7 
~8 
8 
8 
-9 
-l2 
-l2 
«12 
«12 


«12 


-l2 
~12 


~12 


TABLE 8, 
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1lO0-11 12=14 


=4 


=F 


14=15 
=] 


~§ 


<3 

«10 
-12 
-12 
-12 
-12 
«#12 
-12 
-12 
-l2 
“12 


-1l2 


16<17 


-10 
-10 


-12 


#12 
-1l2 
-12 
~12 
-l2 
“12 
=12 
=12 


-l2 





CEILING AYD VISIBILITY 


Lowest Ceiling Lowest Visibility 
Srror in 500 feet Units Error in Miles 
O 1 2 3 4 5 6 0 E225 3am 
® 8 5 OG = Se -i2 8 6 -1 -7 -12 
B 6 4 # =—7 =12 m2 el 6 5 #2 aes -12 
C 4 2 -—3 =-7 #12 =12 ~12 4 3 3 9 12 
D 3 21 -3 -8 -12 #12 -12 5 2 a = a 
m 5 3 2 =] -9 «3-12 =k > 3 = =i 
F 3 #1 -3 -8 -10 -12 -12 3 2 -2 -6 #9 
G 2 0 <4 -3 -10 -12 -12 2 0 43 -7 <9 
mos 7 0 = a -9 =k 4 53 0 =} <=] 
I lL O -2 =4 -7 -9 = -12 1 O =-2 =4 =7 
2 0 =2) 45 2 =10 «~12 1 O -2 -5 4-8 
Mm > 2 i142 - =-7 -9 4 2 9] =4 «7 
em © O -l =3 =6 -=8 -9 lL O —2 <4 «7 
MO -l -2 =-4 -6 -=§ -=10 O =) -3 =-5 -8 
NW O -1 -2 =4 -~6 -8 -10 O -1 -3 -6 «8 
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-12 
-12 
-12 
-12 


-12 


-10 





VIII. CONCLUSION 


As was stated earlicr the final score computed as above has no 
percentace connotation, The highest score attainable is 80, Fron 
he amount of change from previous day reaulired it is very unlikely that 
e score hicher than 40 would be possible. If the woather were cxactly 
the same as the previous day and were correctly predicted the computed 
score would be zeroe 

Several writers have expressed the belief that blind persistence, 
forecasting no change day after day, would give about 50 percent successes 
scored oy fixed tolerance verification. This type of pure persistence was 
tested with the scorins proposed here and the averare score was ~55~6 

In another test a small croun of forecasts were prepared and verified. 
for a selected fsroup of naval stations involvins a wide variety of feo- 
graphical locations. In this test five forecasts were made for the follow 
ing daytime period 0600-1800 and verified with the proposed tolerance tables. 
the averare scores were: Boston -40, Pensacola =-35, Coco Solo ~42, TMonolulu 
=42, San Francisco -39, Kodiak -36, The sample is certainly small ond the 
scores may well reflect the nersonal forecastine experience of the author, 
but it is believed thas they are fairly representative, The averare score 
on these forecasts was -39. The uniformity of scores from hich and low 
latitude indicates that the system has compensated fairly well for var-inc 
difficulty of forecasts. If any bias still exists, it is probably in faver 
Oo: the higher latitude stations with their more difficult, chanrince weoathere 

Tne final determinavicn of the average attainable score and how well the 
system fulfills its primary purpose of measuring skill in forecasting will 


await wider use and tests in a much larger number cf casess 
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Tho proposed tcrecast form is mach Wetter suited to Yaval uses than 


vhe present form because of the separation of Porocast elements and tko 
conpleteness with which they must be forecast in unequivocal termse This 
forecast form will require that the meteorolocist have a clear ricture of 

the expected weather in mind when malzing the Corecast, as tho specifications 
eall for the values of most weather elements to be reduced to numbers stating 
highest, lowest, and average values expected and to state as closely as pos~ 
sible the time of occurrence. Definiteness as to event and tine of occurrence 
diminish as the neriod of time of the forecast increases. However, a fore- 
caster should be able to specify the weather within close limits for at least 
three of the rronosed J2ehour periods, 

Another use of this verification system is in the evaluation of the 
so=called objective or "mechanical" forecasts. As lone as these forecasts 
are used in definite forecasts of an occurrence or nonoccurrence tyne there 
woulce be no significant advantage in usinre this type of verifiecatione The 
type of forecasts made according to inflexible nrocedcures and from predictants 
whose connection with the variable forecast is not knowm should be confined to 
forecasts of a ves = no tynes 

when mechanical techniques are used to predict the value of continuous 
variates, such as wind, sky condition or temperature, any defects will be 
obvious when the forecasts are verified by this proposed system. The low 
scores occurring as a result of the large errors made by a mechanical systen 
will overbalance their successes, especially if the system does not contain 


all simnificant predictants which affect the weather occurring, 
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The chief advantace of the verification system over the one presently 
used is that the final score more nearly reflects only forecasting sill. 
the scores obtained on successive days at the sare stations and those ob- 
tained at different stations are comparable. The verification systen offers 
incentive to the forecaster at both tropical and hirh latitude stations, re- 
quiring definite effort at both places to obtain high scorese The verifi- 
cation system leads tho forecaster to state the forecast as precisely as 
possible, where the present system encourages hedging with indefinite terms 
and uses tolerance tables which favor certain numerical forecastse 

After more use or tests of the rroposed verification system, it may be 
desirable to alter the derree of difficulty in some of the tolerance tables 
by increasing or decreasing certain valuese Further tests may show the de# 
Sirability of a further shift of the zero point from the value used heres 
A possible new reference point is the averase score attained by persistence 


forecasts instead of the score of a correct persistence forecast, 
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